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iBSTHACT . ^ • . 

Course content validation was developed as a 
diagnostic model to ascertain the extend and quality of^ ^ 
implementation of individualized instruction at College of the 
Mainland (COM) • This process^ diagnoses the quality of instruction as 
judged by instructional peers at other institutions. Content raters 
are sent course syllabi^ course documents^ selected examinations^ and 
descriptiotf3 of the course target group and COM student body^ and are 
asked fill out a checklist including questions relating to the 
course rationale^ learning objective.Sy learning -activities^ testing 
techniques^ and course organization. To date^ 27 courses have been 
sent to 135 raters throughout the United States. The '116 yat.ing 
checklists whicli were returned represent an 86 percent response rate.; 

The raters felt that the courses at COM have well-developed * 

J).ehajdJor^al-ob^eativesy---a-nd^thaHt^^ relevant^ 



challinging^ and appropriate for the target group.. Overall^ content 
standjaf ds ^re high^ and* these high standards- are reinforced with 
rigprbtts testing procedures. However^ only 15 percent of the raters 
encountered courses which are open^-entry/exit in format^ and only 11 
percent encountered self- paced courses^ implying alternate learning 
a<:tiyities. T-abulated response's to 24 -of the 27 checklist questions 
are included in this document. (iuthor/NHM) 
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• " College of the Mainland is undertaking a .comprehensive 
effort toward improving instructional effectiveness. Ins true- 
tional improvement is np^t a novel process at the college^. Over 
tHe~last~fiye years "there have beH'mjiier ous "programs^ directed^ 
toward the- betterment .of instruction^. Exterisive .in-service 
training has provided faculty much exposure to the nature, scop( 
and strategies of individualized instruction. 

College of the Mainland has made an institution -wide corn- 
mi tmeiat to individualized- instruction. Briefly stated, instruc- 
tion is individualized when: 

(a) learning outcomes are specified in terms of 
. behavioral objectives ^ - 

(-b-) — s1:udentrs~dan master objectives through ~ 

■ alternate learning activities ; arid, 
(c) students can learn at their own pace (,this 

criterion does not imply that students be offered 
"independent- study''. -Self-^acihg- is- -suggested . 
^Ts a result of research findings which support 
' tlie contention that learning rates vary from 
individual to individual). . 

Despite a long standing commitment to individualized ins true - 
tiot>, the extent of implementation Is' not known. It was to ascer- 
tain the level of instructional development .that diagnostic models 



were designed atid ap^ied, /| . 

One such diagnostic model is course- content validation* 
This process serves to diagnose the quality of instruction, as • 
jju4g3d by instructional peets . • " - . • 

In designing the. strategy for content validation several 
' criteria were considered: 

(1) Content can best be validated by persons • 
having expertise in the Instructional 

,\ specialty being diagnos^ed; 

(2) Objectivity requires that content -vali- 

dation be done by persons outside the ^ 
institution. 



'C31 It .is important that persons selected as. 
' • course content evaluators have a well- 
•developed unddrstaHding of * individualized 
instruction. 



Content, raters were selected on the basis of these criteria 
Each rater was sent the following: 
■^^ course syllabi 
- course documents 
■^'^ selected examinations 
V ^4 description of the course target group 

* a description of the student body at CO. M. /and 
^ a checklist.' ^ 



Devfilbpment of the checklist -was discussed' extensively in 
th^^preface to- the Findings of Round I of course validation, and 
need not be duj^licated here. The checklist contains questions 
relating 'to the rationale, learning objectives i learning activ- 
ities, testing techniques, and organization of the course being 
rated. . ' ' . ■ 

. „ Since individualized Instruction is a rather complex concept, 
a number of items were included in the' r;ating instrument to de- 
tennine the level of an open-entry/ exit, serf-pa;ced instrxiction. 
The rating instioiinent contains twenty-seven questions. Twenty 
four questions require structured responses, thip^e permit i^pen- 
ended response. 

, ♦ > ' 

— 5o-d^te7-±wo-Tounds of""C"onr^s^have been rated for content 

and organization. In Rounds I and II, twenty-seven courses were 

sent , to^ approximately 135 raters, throughout- the 4Jnited States : - - - 

116 fating checklists were returned. This represents a very 

favorable response rate pf* eighty-six per cent/ 

7 y ^Follwitig is a question by questipn description and discusafion 
of the rating results. _ • 



ROUND I 



Auto 165 - Auto Air Conditioning 
Biology 141,. 142 - General Biology 

Business Communications 231 - Business Communications 
Economics 231 - Principles 

Electronics 141 - Elfectt^c^l Electronics Circuits I 
English 131 - Composition & Rhetoric in Communication 
History 135, 136 - U. S. History- 

Nursing 161 - Physical and Mental Health and Illness 
Nursing 262 - Physical ^nd Mental Health '^nd Illness 
P. E^.ill - Foundations of Fitness and Health 
Psychology 23r- Introductory 



ROUND II 



Art 13 1^ 132_- Basi.^ Desig|i_ I^JI • . - _ -^.^ 

'CHeiSistry~I4T, '1\2 - Geneirar Inorganic 7 
Cooperative Education 

Englisl\ 132 - Composition and Reading ' ^ 
Introduction to Business 131 - Introduction. ' 
Management 234 - Principles 
„Marketing-»231 - Principles 

'Math 231, .232 - Calculus I, 'II ^ \ ; 

P. E. 117 - Beginning Tennis • ^ 

Political Science 235, 235 - American, National J and State Government 
^Sociology 131 - Introductory <> " 

Speech 131 - Fundamentals of Speech - <i 

Psychology 233 - Child Growth and Development 



. I ' ' At • » ^ 

QUESTIONS . ' ^' 



V 



1. Does the syllabus attempt to justify the course content tb the 

^ /Student? ^ ' 
, . VALUE ^ CODE " PERCENTAGE 



" No justification 1 '2 

« " ^^^^^ • < 

Excellent justification 5 33* - 

«> ' , Raters were, asked about the .extent to^which course content 
was justified for students. Generally, it is recogni?;ed that a* 
rationale provides pos'itive inducement for students tcT' undertake 
the learning process. Eighty-nine per cent of thfe raters assess- 
- ed cour a^^s- which r p r-e-vjrded ^vex age and abb^^e averag^. justification 



of;^ course content to student^. ' / >^ 



2v "Are -the outcomes referred to in .the objectives specific and ~ 

measurable? .1 , - * 

•<&'." tf. ' . 

VALUE > ' ^ CODE PERCENTAGE 

Vague, and Unmeasiirable 1 ^ 7 ' - . ' 

■ 2 -. 7 ' 

' " ■ . '3 4 

^* 4 - .32 

^Specific "and Measurable 5 ""SO- 
This question sought to ascertain the extent to which learn- 
^ ing- outcomes, (not necessarily ••objectives") were specific and 
.measurable. Well-written course objectives should describe term- 
inal behaviors ^which are both specific « and measurable* Eighty-six 



peY cent (eight out of ten^of the rat<^,rs) encoun^red courses 
averages and above average in specificity and measurabilit^* ^ 



9ow would you desjiribe the tferminal cop[ipetencies of .the course? 

V^iue Code Percentage ^ " * ' <f 

Simple , Rote, Memorization 1 ' 1 * ' 

nLa eking challenge ' 2 4 ' , ... 

■ • 3 .21 • . 

, . ' 4 ■ 46 ■ ■ ' . " 

Challenging, requiring ^ 

Complex Mental Skills 5 ^ 28 " 

Difficulty of terminal competencies wa^ ra|:ed in this question. • 

Raters were asked to judge the quality of Objectives* content Only 

five per cent encountered competencies which were considered simpl^e, 

rote or unchallenging. The remainder rated courses which were average 

or above average in requiring c^ornplex^i^^^^ - 

to thfe learner. 

. ^ 

Do objectives follow a logical sequence from ease to difficulty? 
Value Code Percentage 

No .1 6/ ' - ' ^ " 

■ ■ 2 ^ 6 - 

3 lA ' 

. . , . 4 38 ........ 

Yes . . ' 5 36 ' . ' ' ' 

A well-constructed course should ideally progress |rom a start- ' 
ing point of relative ease, to more sequentially difficult taskff.. 
In this fashion, initial su'ccess is likely to be experienced by 
Students As students develop more positive ^eel^ingsi toward course 



content, thky will in\all likelihood be better, prepared for more' 
difficult cognitive objectives later in the course. Eighty-ei^ht^ 

per cent of the .raters encountered courses that have average. or 

* * . ' . * " / , " ' 

above average organisation ^long a progress iqp from ease to difficulCyv 



Is there, any effort to mal^e the objectives relevant to tKe*studenV? - 
Value * Code Percentage 



No/ ^> 1 * . ,..6 

■ . ^ 2 . J.5 

j> ■ • • , 

3 '13 

4 ' li\ 
Ye? 5 • , ■ 42. 



Numerous , veil-supported arguments have beefi* advanced to sug-^ 
gest -that learning is more likely to occur when objectives attain 
relevance to students. Seventy-nine per cent of the ratera evalu- 
ated courses average and above average in.»terms of offering rele- 
vance to students. . - 



Given the studentJ:Lar get group, would you use the same objectives 
if this were your courseT ;~ 

Value ^ Code Percentage * ' . , 



None ,1 • '5 

2 * 10 

3 27" 

4 . • ' . . 42 
All 5 . . ■ 16 



. In order that, .'multiple- assessments of each element of 'the 
course be obtained, raters were also asked "to judge whether they 



would .use similar objectives for a similar course. Sixteen per cent" 
Wd use "All". Fi-ve per ..cent woul'd use "n^ne'.': Eighty-five per..^ 
cent rated, objectives average -andvabbv* average in tefms of adoption 
for similar courses an.d -target grouj).. . ' " 



li^^ ^^'^f-'^^ mastered all requirements of this coufse with an "A" ' ' 
?5Ssf '^""^"^^ assign ^ifstudents performed ' idlnScall-?- in your 



Value 


' Code 


F 


' •' ■ 1 


D ♦ 


2 • 


C 


.3 


B - 


4 


A . 


. 5 



Percentage 



0 
0 

■5. 
1 

.94 



A relatively broad oontent assessment should derive from a 
judgement as- to how well , V "A" student in the course being rated 
woufd do -in the rater's course. Raters .would award sti^ents "A's'- 
at. ,a level approaching statistical significance' at' .05 probability. 
Ih other words, there is substantial probability thit raters would 
not awardf less than an ''A". - . 



If a student mastered the course objectives would he/she be pre-, 
pared, for . the next course in sequence (if ' any)-.at_3:our_conege? 



Value Code 



Percentage 



No ■ 1 V . ■ p. 

\ 2 ' . 1 

> •.. • ' 4 _ 22 

Yes 5" • 70 



The^ extent to which the course under scrutiny would prepare 
a student for more advanced- classWork was evaluated.^ Ninty-two 
per cent of raters judged sti^dents feo be above . average in prepar- 
ation for subsequent courses. • . 



Would a senior college accept competencies required, here 'as ejquiv- 
aleht o-f its,.ovm co^r^e, if applicable? 

' Value - Code .Percentage • 

. . • X No . 1 . 4 . 

■ '2 2 " 

% . • . 

Yes 5 - . 67 

♦ •. . • ' ■ . • 

Since each .question of the ins trjument -might possibly be' In- 

Jierpreted out of 'context, multiple measures, of course content 

quality vT^ere* sought. v * * ^ v " ^ . . 1' , 

. "Ninety-four per cent of raters 'felt pourses' would serye 

students well in comparison to similar senior college courses. 




I^at is your overall evalu^ation^ of the content standards described 



Value ' Code Percentage 

Poor, very low standards 1 . . 1' 



2 . . 2 

3 10 



• ■ . 4 35 ■ ■ . 

^Excellent , 'high standards 5 • 52 . . ^ 



Affinal question* axldressing the quality of course content 
dealt with overall impressions. Dn a cofttinuunu flanked by poor ^ 
lov standards; and- e;ccellent,^ljigh standards, 97 per cent of ' 
the raters judged bourses avera|5;i6 aivd 'above average* in terms of 
excellence. • . • * . ' i 



•Could a student ^enroll or complete ' this* course at any^ tijne? 

. • Value ' . Cpde Percentage , . 

Yes ' i ' ^ 15 ^ ^ J " ' 

NO; 2^ . '.47 \ , 

.••>.. ■ - 

Not- Indicated ' " 3 - <- "SS 

Course*^ organiz'^tlon provides a measure qf whether or not 
ind'^yidualization has occurred. Ideally, i^^a truly . individualized, 
^course, students should be able to progress at their oWn xate of 
learning. This is not to imply that ins'truct ion need be ^^independ- 
ent study/* Community 'college students are often unable to gener- • 
atfe niotivation necessary td carry them' -through' a'semester, of^loosely 
non-supervised activity. However, if it is poi^sible -for^ a student . 
.to enr6ll and complete -a course" at different paints in a semest^t, 
without penalty > ,then some instructional development 'could be said 
to have occurred. " 

Fifteen per cent of raterjs en^countered courses which pernvLt 
open entry-exit; 



> 1 . • 

Could a student take as long as he/she wanted, -and sti^l complete , 

the course with an "A"? '< "^i^ 



Valu^ ' • Code Percentage 

.■„. , Yes-' ^< ------X- • - - • i-x- _ ' - V 

■ •= No - - — 2 ■ ' ,• 55 ■ 

. . , , Not indicated . 3 ' * '34 

If students are tmable to complete a course on time, they 
should be allowed more time to do so without penalty. Eleven 
per cent of raters' evaluated courses which allow students to 
extend past semester time limits and still receive an''"A*\- 



Are alterna£.e learning activities provided So that students could 
complete the course without attending leqtures? 

Value Code Percentage - ^ 

Yes 1 19 - , ^ 

, . -fe, No 2 > ' 56 . 

Not indic4ted v' A' 3 r ' ^ 25 > . \ 

About' one in five raters-eWomtered coutses whidh offer studenus 

^altemate- learning activities to lecture. This question is rather jun-r 



qualified - asking whether or^xiot a CQurse'""coglii^e-compl-e 



attending what may Interpreted ^as-a single JLecture,. It was intended 
to ascertain the overall extent to which lectures wei;:e the sole in- 
instructional vehicle. ' * ^ ' 



Do learning activities pre^^cribe active passive leairner^^rbles'J 

. " : Value , Code : » Percen tage 

Active. ' 1 ^ 45 

. . • '3. 19 

, I - 4 • .. • 10- ^ ■ 

' * ' . • • Passive ' 5 . 4 



-To the extent that ''good*M.nstruction is knoxm to occur, a 
studentlslio.u^^ fehafi :pas*slve% role irT the 

learning process. Eighty-six per cent of the raters felt courses 
to be average or above Average in prescribing that students play 
,an active learner's rale. 



According to the '^syllabus media are used: 
Value 
Nfever 



Code 


.Percentage 


1 


7 


2 


9 




27 


4 


43 


5 


' ' 14 . 



, - ■ With Frequency 

' Media use is regarded as a desirable concomitant of individualized' 
instruction. Seven of one hundred of the raters judged courses which' , 
never use media. Elghjiy-four per cent of thfe raters scored courses' 
average and above average in frequency of inddia use. 



Could a student select from alternate learning activities to master 
objectives? " - • • 

i ~ * " » 

Value ' Code Percentage 

Yes 1 . 48 

• No , 2 .. • ' 34 . • 

Not Indicated 3 18 . • . . 

^ On a less global scale, fifty per, cent of raters encountered • 

courses which offered al^ternate learning activities for mastery 

bf '^bme dbjectives. ^ * « 



17. Do* learning "^activities provide opportunities to* master the specific 

J — tasks? - ^ 

Value * Code ' Percentage 

No • 1. . 6 ' . * . . 

\ 2 3 ' ' 

. ' 3 , ^ ' Xk \ 

4 29 • 

- ■ , -•- . - 

• ' Yes 5 48 - • , 

Appropriateness" of learning tasks, in light of objectives was 
evaluated. Only nine per cent of raters scored courses below average 
on this* issue. Seventy-seven per cent scored above average. 



18. Do tests require students to put learned jpaterial together in new 

<^ 

, ways? \ ' ; ' . : . ' 

^ , . Value Code ^ Percentage - » . 

No ^1 ' 14 ' ^ , / . 
Sometimes 2 , ' " • 56 ' 
Yes 3 - 28 



White objectives may be of very high quality in structure and 
content it ds equally important that tests provide an accurate diagosis 
' " of the extent to which objectives are achieved. Objectives might 

resound of "challenge , complexity and higher order mental skills, but ,, 
without the evaluation system toWasure mastery, they may be hollow. 

One index of test quality is to ga^ge whether tests re^juire 
students to deal with concepts in -a different manner from voiilch they 
Were learned. Eighty-six per' cent of the raters encountered courses . 
-in which tests did require recomposition. 

-13- 



Dp tests require students to answer questions. not specified in 1: 



^objectives? • '-4, „ _ J 1. 

Value Code Percentag e 

No 1 ' 59 

Sometimes 2 27 

• Yes -3^ . ' 14 ' 

About four out. of ten raters fell. ' questions went beyond 
the scope of objectives. 



Ate- there enough' test items to evaluate mastery of the objectives 
in each unit? 

' ' > ' . Value ^ . Code Percentage' 

c. No , - 1 ■ 10 '* . 

Sometimes ; .2 '23 

.Yes 3 . 67 

Only ten per cent of the respondents felt there were not 

enough test items to evaluate mastery of objectiyes,. 



Are the test items geared for the level of competency called for 
in the objectives? ' , . • 

' ' Value Code • Percentage ' • 

■ ^ - No ^ 1 , ' 3 . 

Sometimes ' ^ . 2 • 12 ' 

Yes 3 • " .85. •• ■ 



• Tesf Ttems 7 according to raters , are geared for the level of 
competency called* for- In objective's .~ A'ffirmative responses are- 
probable at the .05 level of confidence; 



Given the same objectives, ,how would students in your class'" consider 
the test items? 

. * Value 



Extremely Easy 



Code . 


Percentage 


1 


0. 


2- 


7 


3 


49 


.■4 




5 


' 7 



. ^ Extremely Difficult 

One would not expect a rater to judge that his/her. students 
would find test items "extremely difficult.'' Such a statement 
might sound ^as if the rater's students' were not up to par. However;, 
seven per cent did rate questions as extremely difficult. Ninety- 
three pi^r cent ratted test items average and above average in diffi. 
culty, and none asserted test items were "extremely easy.'' 



G ven the course objectives, would the student target. group be able ' 
to complete the test in the. time allottel? ' v ' . " 

Value ^ Code , ^ PerperitaRe 

* ' No ^ I ^ 2 ^ 

Possibly ■ 2 " . '38 

^ t. . 

. . Yes .-. > 3 ■ 60 . ■ , 



Two per" cent of the raters felt tests did not give students 
adequate time for completion. <rhe remainittg ninety-eight per cent 
responded "possibly" and "yes*'. ' ^ 



24. Generally, how would you describe the- test items? 

Value 



ERIC 



Very poorly written 



Code \ 


Percentage 


■ 1 


3 


2 


. 4. 


3 


18 


4 


54 


5 ■ 


" 21 



Extremely well-written 
Raters^were asked t,o render an overall evaluation of ^ test items. 
On ^' .continuum bounded by "very poorly .j^ritteh", three per c\nt found 
poorly V7ritten tests, .twenty per cent found tests to be extremely 
well written. Ninety-seven per cent test items average and above ^ 
average . , " ' ^- — • 



Questions 25, 26 and 27 requested an open-ended response from 
raters. Raters were asked which objectives should be deleted, and 
which should be added. In addition,* broad and general comments as 
to the overall quality of the course v/ete solicited. These coiranents 
would se'rv^ only. a limited, anecdotal purpose in the overall summary . 
.They can be perused verbatim on individual course evaluations.. 
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SUMMARY, 



Courses- rated^ in Rounds T and II have well -developed behavioral 
objectives • Cou;^se objectives rejjeiyed faypr^^ 



of relevance, challenge, and appropriateness for the target group* 

Requirements for courses rated in^ Rounds I and II are demanding • 
"A" students at C,0,M. vould be "A" students' elsewhere. Students 
completing .C,0,M, courses could be expected to do well at universi- 
ties and other Comnfunity Colleges, Overall, content standards re- 
ceived excel/ent ratings. Only three it) one hundred raters encoun- 
tered courses ^below average, in^ overall quality. ^ 

Gonteni is one element, instructional delivery techniques are 
another. Only fifteen -per cent of the raters encountered courses^ 
.whicli are /open-entry* /exit in format, *Even fewer, eleven per pent, 
rated courses which permit , students to learti /at their own' pace, with- 
out penalty, ' " . ^ * 



A necessary prerequisite of open entry/exit and self-paced 4.n- 
^S'truction is that alternate learning techniques be made available to 
studehtSy while some portions of courses can be acpompli^hed with 
alternate activities, lecture is the ptedominant instructional de- 
livery/strategy.. Extensive use of media is not apparent from the 
materials supplie^d to ratejrs, - , . 

\Tests do not appear to be incorigrous with pbjectiyes , 'Raters 
found Lest items to be consistent withv terminal behaviors specified 
In course objectives, ■ • ' j . 



Ratings received in Roiinds I and II yield several ^conclusions 



* . (1) 'Course standards are demanding to the extent that 
thejc constitute more than adequate preparation 
for transfer. \ 

(2) High .standards are reinforced with rigorous 

. »• 

testing procedures. 

l(3) , The area least favorably^ evaluated cbncemed class 

structure permitting alternate learning activities and 
^ * self «pacing;--., .-^^ — ; ^" " 



The final conclusion in light of' the generally excellent 

i ♦ - ^ * * 

evaluation of other aspects' of structure, content, and method- 



ology, requires further elaboration and an attempt to explain 
the discrepancy. One factor may be purely semantics. That is, 
the term ^'behavioral objective'* has been clearly defined (by 
Mage r and others) and has been used to the extent that its 
meaning approaches universality. However, "self-pacingV lacks 
the clear definitional parameters v;hich characterize objectives'. 
It may be interpreted as complete student independence fr^om 
faculty guidelines, or it may represent re-cycling or re-testing 
within units of varying lengths. 

The 'p^resence{,or absence of lectures as components of an 
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individualized course of instruction has frequently been a point 
of contention. In Postlewaithe* s. early work on audio-.tutorial 
biology, a pioneer effort in -self-paced instruction, his first 
major revision was to reinstate l|oth general assembly /sessions 
and small group discussions. Sutlpequent research has supported th 
need for initial lecture sessions! to oriont the students to the 
use of' an individualized fonrfat. Ij ' • . ' . ' 

Therefore, the results of outside evaluation to date should 
serve as signals, suggesting thatj self-pacing and alternate 
learning activities should be exapned, but it should not be 
accepted as definiy^_ey£dence--th'a1r-these components are below. 
■P'^'^^ Npt only should the components be examined, but attention 
should also be afforded to the definition of these terms in the 
course documents, and to the projiortion of lecture and alternate 
activities. selected for inclusion in the courses. 
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